Acquiring Our Data

In the world we live in today, data is very valuable. Big companies such as Google, Amazon, and Facebook make much of their income from analyzing and selling data that they acquire from people that use their products and surf their sites. This acquired data provides valuable information; what interests people have, what their purchasing habits are like, what items they are looking to purchase, etc. Acquiring data, therefore, is a task of great value. If we are able to extract data from websites and analyze the information this data provides, we will have a powerful product in our hands. To begin with, we will extract data from Amazon, the worlds largest online retailer and therefore a great place to find valuable information from a multitude of products as well as countless customer feedback. However, extracting this data manually will without doubt serve to be a monumetal task. Luckily, there are ways we can automate this data-extraction process and store our information in a secure database.

How?

How we did it

The process of data collection is one that can become very complex if not scaled properly. Since we have the goal of allowing data collection and processing to be a manageable task for everyone, regardless of the size of their resources, we must be able to provide a service that is able to do so. We want to be able to collect product information from an online retailer and store it in a database where we will be able to extract it later for analysis.

How can we do this? The answer to this question is web scraping. Web scraping is the process of retrieving data from a website. You might ask yourself: "I can collect data from a website myself, why do I need a web scraper?" The answer to that question lies in the fact that a web scraper allows us to automate this process. By automating the data collection process, we are able to scale our application in a way we would not be able to if we were collecting data manually.

The benefits of using a web scraper

We use web scraping for a variety of reasons. For one, web scraping is a process of automating the extraction of data in a quick and efficient way. Through a web scraper, we can extract data from any website (regardless of the size of that data) and store it safely. Additionally, websites may have data that is not able to be copied and pasted the way we normally do when manually extracting information from a website. However, with a web scraper, this is no issue. Another reason we use a web scraper is because of its flexibility. Web scraping allows us to extract information in our format of choice, whether it be a JSON file, a csv file, or many others.

For the purposes of fulfilling the goals we have with this product, a web scraper makes perfect sense. We want to collect vast amounts of data in a way that is efficient and accurate. Due to the fact that Amazon is such a large online retailer filled with an immense variety of products and information, we have decided to scrape our data from their site. However, the power of web scraping lies in the fact that it is flexible and allows for data collection from any website. How can this be? What is happening on the back end that allows a web scraper to do what it does?

How does it work?

The basics of a web scraper are relatively simple to understand. The first thing we must understand is that the "language of the web" is Hypertext Markup Language, or HTML. HTML is the standard markup language for web pages, which are composed of building blocks called HTML elements. In turn, HTML elements are represented by <> tags. Every part of this webpage you are seeing now, from these words to the header above to the images you see are all built using HTML tags. These tags have, in most cases, what we call classes and ids. These classes and ids can be used to identify HTML elements, such as the name, price, and rating of an Amazon product. If you want to learn more about the structure of HTML elements and how webpages work in general, this page can serve as helpful tool.

To be able to understand how web scraping works, we also have to understand its components. In essence, web scraping works by way of two parts: a web crawler and a web scraper.

Web Crawler

A web crawler (also known as a "spider") is an ai that browses the internet and searches for content by following links and exploring. Usually, this is done using indexing. The spider will visit all the specified pages of a website and find links to get the required data. In our case, the spider visits the Amazon homepage and searches for a term specified by the user. It then looks for all the links to products that it can find. If you want to check out the way we coded our spider, feel free to visit our GitHub repository. A link to this repository can be found here as well as in the button down bellow. The spider was coded using Python and the file can be found following this path:
ws seniorproject -> spiders -> testfile.py


Web Scraper

A web scraper, on the other hand, is a tool that is designed to accurately extract data from a webpage. Web scrapers automate the data extraction process, allowing for rapid extraction of data from a website. This is done by specifying which HTML elements we want the scraper to extract. For example, if we wanted the scraper to extract the header above this paragraph, we would tell it to look for the unique HTML element combination of tag "h3" and id "web_scraper_header". This is because our header tag looks something like this:

<h3 id = "web scraper header"> Web Scraper </h3>

While this may seem confusing at first, acquiring a basic understanding of how HTML elements are declared will allow you to follow along with our code and even make changes of your own.

Feel free to visit our GitHub repository where you'll be able to find our code. Additionally, feel free to clone or download the repository and check out the scraper on your own computer. Follow the steps laid out in the "readme" file to download the dependencies and run the scraper. For this project, we used the web crawling framework known as Scrapy. If you want to know more about this framework and how it facilitates the web scraping process, visit this link.

In the following section you will be able to see some of the datasets we scraped using our custom web scraper.

Check out our code on GitHub

Use the follwing dropdown option to select one of our already scraped datasets. These have all been collected from Amazon using our custom web scraper. Once we have collected datasets such as the ones you can choose here, we will be able to analyze them and extract patterns using our machine learning and ai models. For now, you are free to inspect the data yourself. The products selected here are not biased; we simply ran our scraper and input shoes/pants/shirts/hats and the following is what it returned. One of the things you may notice is that the scraper returned around 15-20 products. This was done intentionally as we did not want to overload the user; we simply wanted to offer a preview of the scraper's functionality. Also, the attributes you will see (price, rating, etc.) are not the complete list of attributes our finished scraper includes. The reason for not including all the attributes is the same reason as before: we did not want to overload the user, only offer a preview of the scraper's functionality.

{% if product is defined %}

{{product}}

{% endif %}
{% for col in colnames %} {% endfor %} {% for record in records %} {% for col in colnames %} {% endfor %} {% endfor %}
{{ col }}
{{ record[col] }}

On to the next...

Now that we have a clearer understanding of how we collect our data, it is time to understand how we will analyze it. Machine learning and ai is an enourmous and relatively complex field filled with algortihms and problems that the data science world has as of yet fully mastered. We hope to bring you a better understanding of what machine learning actually is, why it is useful today, and why we believe it is the centerpoint of the next industrial revolution, one we are already living.

Follow along